2/n Enable 16-bit activations and 8-bit weights in Cadence Quantizer for linear #15901

RahulC7 · 2025-11-19T22:21:45Z

Summary:

Context

We continue from D84284794 to add support for 16-bit activations. Note that right now, all though they support 16-bit activations already, it's only if the weights are also 16-bits. To do this, we need to change the way we template some functions.

Current Behavior

Right now, we're composing two macros together, the ET_FORALL_JARVIS_QUANTIZED_TYPES_WITH_INT16 macro:

https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/operators.h?lines=22-25

and the function macro(quantized_linear chosen for example):

https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/quantized_linear_out.cpp?lines=30-41

so together, it just becomes a switch statement, calling the quantized_linear function with the correct template parameter.

However, note that it assumes that both the input activations and weights are the same dtype, which is not the case.

This Diff

We fix the generic implementation by allowing there to be two generics, one for the weight and one for the input activations.

Reviewed By: hsharma35

Differential Revision: D86538176

pytorch-bot · 2025-11-19T22:21:48Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15901

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 1 Unrelated Failure

As of commit 40d03e1 with merge base d2c011e ():

NEW FAILURES - The following jobs have failed:

pull / unittest / linux / linux-job (gh)
exir/backend/test/test_lowered_backend_module.py::TestBackendAPI::test_emit_nested_lowered_backend_module
Test CUDA Builds / check-all-cuda-builds (gh)
Process completed with exit code 1.
Test CUDA Builds / test-executorch-cuda-build-13.0 / linux-job (gh)
RuntimeError: Command docker exec -t 3f238a5675f4bd84317093afa6cb144ca99cf5066e3d3aa9c92ed77b335e09e4 /exec failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / unittest-editable / linux / linux-job (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

meta-codesync · 2025-11-19T22:21:54Z

@RahulC7 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D86538176.

github-actions · 2025-11-19T22:22:38Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Copilot

Pull Request Overview

This PR adds support for 16-bit activations with 8-bit weights in the Cadence Quantizer for linear operations, continuing from a previous differential (D84284794). The key changes enable mixed precision quantization where activations use int16 while weights use int8, addressing the limitation where both had to use the same dtype.

Adds int16 activation + int8 weight support by delegating to generic implementations from Jarvis/min_runtime/operators
Introduces new build dependencies to support the int16 templating
Adds test coverage for int16 activation scenarios

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
backends/cadence/hifi/operators/op_quantized_linear_out.cpp	Adds conditional branches to handle int16 activations with int8 weights by calling generic implementations, and removes `__ET_UNUSED` markers from now-used parameters
backends/cadence/hifi/operators/targets.bzl	Adds dependencies on Jarvis min_runtime operators to support int16 templating and removes `quantized_linear_out` from the main operator list
backends/cadence/hifi/operators/tests/test_op_quantized_linear_out.cpp	Adds new test file with int16 activation test case for `quantized_linear_out`

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-19T22:26:25Z

backends/cadence/hifi/operators/tests/test_op_quantized_linear_out.cpp

+using std::string_view;
+
+class HiFiQuantizedLinearTest : public OperatorTest {
+ public:


[nitpick] The public: access specifier is redundant as there are no public members declared. Consider removing this line to improve code clarity.

Suggested change

public:

Copilot · 2025-11-19T22:26:26Z

backends/cadence/hifi/operators/op_quantized_linear_out.cpp

+      out_zero_point,
+      offset,
+      out
+    );


Missing return statement after handling the int16 activation case. The function will continue to execute the subsequent else if branches even after successfully handling the Short/Char case, potentially leading to incorrect behavior or error messages. Add a return; statement after line 236.

Suggested change

);

);

return;

Copilot · 2025-11-19T22:26:26Z

backends/cadence/hifi/operators/op_quantized_linear_out.cpp

+      out_zero_point,
+      offset,
+      out
+    );


Missing return statement after handling the int16 activation case. The function will continue to execute the subsequent else if branches even after successfully handling the Short/Char case, potentially leading to incorrect behavior or error messages. Add a return; statement after line 295.

Suggested change

);

);

return;

Copilot · 2025-11-19T22:26:26Z

backends/cadence/hifi/operators/op_quantized_linear_out.cpp

    int64_t out_multiplier,
    int64_t out_shift,
    int64_t out_zero_point,
    __ET_UNUSED const optional<Tensor>& offset,


The __ET_UNUSED marker should be removed from both the ctx and offset parameters since they are now being used in the int16 activation case (lines 284, 293). This creates an inconsistency with quantized_linear_out (line 211, 220) where these markers have already been removed.

Copilot · 2025-11-19T22:26:27Z

backends/cadence/hifi/operators/targets.bzl

        define_operator(op)
+
+    # quantized_linear_out and quantized_linear_per_tensor_out needs additional dependency for int16 support
+    define_operator("quantized_linear_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers",])


Remove the trailing comma after the closing bracket on line 126. While Python allows trailing commas in lists, this is inconsistent with the style used elsewhere in the codebase and may cause issues with some linters or build tools.

Suggested change

define_operator("quantized_linear_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers",])

define_operator("quantized_linear_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers"])

Copilot · 2025-11-19T22:26:27Z

backends/cadence/hifi/operators/targets.bzl

+
+    # quantized_linear_out and quantized_linear_per_tensor_out needs additional dependency for int16 support
+    define_operator("quantized_linear_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers",])
+    define_operator("quantized_linear_per_tensor_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers",])


Remove the trailing comma after the closing bracket on line 127. While Python allows trailing commas in lists, this is inconsistent with the style used elsewhere in the codebase and may cause issues with some linters or build tools.

Suggested change

define_operator("quantized_linear_per_tensor_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers",])

define_operator("quantized_linear_per_tensor_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers"])

…for linear (pytorch#15901) Summary: # Context We continue from D84284794 to add support for 16-bit activations. Note that right now, all though they support 16-bit activations already, it's only if the weights are also 16-bits. To do this, we need to change the way we template some functions. # Current Behavior Right now, we're composing two macros together, the `ET_FORALL_JARVIS_QUANTIZED_TYPES_WITH_INT16` macro: https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/operators.h?lines=22-25 and the function macro(`quantized_linear` chosen for example): https://www.internalfb.com/code/fbsource/[9e8c6d8466107f58aa3de1b9e4ec71c49d670a8f]/fbcode/on_device_ai/Assistant/Jarvis/min_runtime/operators/generic/quantized_linear_out.cpp?lines=30-41 so together, it just becomes a switch statement, calling the `quantized_linear` function with the correct template parameter. However, note that it assumes that both the input activations and weights are the same dtype, which is not the case. # This Diff We fix the generic implementation by allowing there to be two generics, one for the weight and one for the input activations. Reviewed By: hsharma35 Differential Revision: D86538176

Copilot AI review requested due to automatic review settings November 19, 2025 22:21

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 19, 2025

meta-codesync bot added fb-exported meta-exported labels Nov 19, 2025

Copilot started reviewing on behalf of RahulC7 November 19, 2025 22:22 View session

Copilot finished reviewing on behalf of RahulC7 November 19, 2025 22:25

Copilot AI reviewed Nov 19, 2025

View reviewed changes

RahulC7 force-pushed the export-D86538176 branch from 9e4a0d9 to 0bfea2e Compare November 20, 2025 21:55

RahulC7 force-pushed the export-D86538176 branch from 0bfea2e to 40d03e1 Compare November 20, 2025 21:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

2/n Enable 16-bit activations and 8-bit weights in Cadence Quantizer for linear #15901

2/n Enable 16-bit activations and 8-bit weights in Cadence Quantizer for linear #15901

Uh oh!

RahulC7 commented Nov 19, 2025

Uh oh!

pytorch-bot bot commented Nov 19, 2025 •

edited

Loading

Uh oh!

meta-codesync bot commented Nov 19, 2025

Uh oh!

github-actions bot commented Nov 19, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Nov 19, 2025

Uh oh!

Copilot AI Nov 19, 2025

Uh oh!

Copilot AI Nov 19, 2025

Uh oh!

Copilot AI Nov 19, 2025

Uh oh!

Copilot AI Nov 19, 2025

Uh oh!

Copilot AI Nov 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	define_operator("quantized_linear_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers",])
	define_operator("quantized_linear_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers"])

	define_operator("quantized_linear_per_tensor_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers",])
	define_operator("quantized_linear_per_tensor_out", deps=["fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:quantize_linear_out", "fbcode//on_device_ai/Assistant/Jarvis/min_runtime/operators:headers"])

2/n Enable 16-bit activations and 8-bit weights in Cadence Quantizer for linear #15901

Are you sure you want to change the base?

2/n Enable 16-bit activations and 8-bit weights in Cadence Quantizer for linear #15901

Uh oh!

Conversation

RahulC7 commented Nov 19, 2025

Context

Current Behavior

This Diff

Uh oh!

pytorch-bot bot commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15901

❌ 3 New Failures, 1 Unrelated Failure

Uh oh!

meta-codesync bot commented Nov 19, 2025

Uh oh!

github-actions bot commented Nov 19, 2025

This PR needs a release notes: label

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pytorch-bot bot commented Nov 19, 2025 •

edited

Loading

This PR needs a `release notes:` label